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INVESTIGATIONS OF THE DIAGNOSTIC RELlABILrTY OF 
. READING SPECIALISTS, LEARNING DISABILITIES SPECIALISTS, 
AND CLASSROOM TEACUERS ^ RESULTS AND IMPLICATIONS 

Annecte B. .Welnshank" 

The studies raported in this paper were designed by the Clinical 
Studies Group of the Institute for Research on Teaching to Investigate 
. the clinical ^roblem^solving skills o f reading and learning disabilities 
specialises and classroom teachfirs as tiity diagnosed, and then proposed 
reraediations' for, a variety of reading problems. The mode^l for all 
these studies of clinician diagnostic perfbrmance was an Observational 
Study, conducted in 1577; of a selected group of eight highly trained 
and experienced reading specialists. The procedures developed for that 
study were used, with some modifications, (Gil, Hoffmeyer, Van Roekel, 
& Welnshank, Note 1) in all the subsequent investigationsr 

The 1977 Qbservational Study 

Desig n and Analysis 

% _ 

Each of the eight clinicians in . the 1977 study was randomly assigned 
to a simulated case of reading difficulty. A elmulated case is a 
collection of information about a child with a reading problem, ' 
The four simulated cases developed by the Clinical Studies Group were 
bassd on real children in grades th^ee through seven Mio' had attended 
the Michigan State University Reading Clinic and were corisidered 'to be 
representative of reading problGms Commonly encountered in public sdhools. 



Annette B, Weinshank iii a teacher co^investlgator with the Clinical 
Studies Project and a former research Intern with that project. She is an 
experienced reading specialist working with the Lansing, Michigan public 
schools. She holds u Ph.D. In Educational Psychology from Michigan State 
Universltv. ... * 



Each simulated case was k^pt in a largta file box, A cue inventory 
was provided listing the information (cues) available for each case* 
achievement Casts, family and academic background, cognitive ability, 
group and individual reading diagnosfedc measures, classroom information, 
work sampies, and so on. The information was presentea in a variety of 
,forfnats: test booklets, audio tapes, examinar-s comments, and test scores. 
Each simulated case also had an equivalent form-^^a superficially disguised. ^ 
replicate of the original, prepared by making minor changes In the data 
base and randomly reordering the cue inventory (Lee & Weinshank, Note 2). In 
addition to these' eight cases (four originals and four replicates), two 
learning disability cases were subsequently developed, and then two 
cases of reading difficulty that used only materials available in any 
ordinary classroom. 

In all observaCional sessions, the task for each of the eight clini- 
cians was to looVJ at case infonnation and write up a diagndsis and an 
initial remediation plan. Each clinician participated in three obserVational 
sessions, with sessions one Mnd three separated by a minimum of one week- 
In the first session, the clinicians worked with one of the four original 
simulated cases. In the second session, they worked with one of thy re^ 
maining three simulated cases. In the third session, they worked with the 
equiva-lpt form (replicate) of the simulated case they had worked with in 
the first session. There ware 24 sessions in all, six sessions per case, 

During each observational sessions the clinician requested items 
of information from the cue inventory. These were handed to the clinician 
one at a time by the experimenter, who recorded the cues ^ requested. The 
experimenter managed the observational sessiona. by providing all necessary . 
materials, timing the various tasks, and audio^taping the proceedingf . * 
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Whtiu the period allocted for cue CDllection was over, the clinicin 
given a fixed amount of time to write out their diagnostic Ju(* 
suggest an initial remediation plan. They were allowed to V 
cues they had requested and any notes they had made while . 
the cues, ' ^ , - 

'The written diagnoses, resulting from the observatior. 
as the unit 'of analysis for this study. The diagnoses wt t 
determine (1) group agreement and (2) agreement between .er- 
correlation) and between each clinician and him/hersalf tion) 
on the diagnostic statements believed to characterize eac 

Extent of group agreement (the proportion of clinician i>,Leeing on 
statements seen as characterizing each case) was measured using a pro= 
portional' agreement statistic. Since there were six^ sessions devoted, 
to each casBj any given statement could be mentioned from zero to six 
times. If a statement such as "instant sight word recognition low" ' was 
mentioned in three sessions^ for examples the proportional agreement "for 
that statement would be .50; that tSj half the clinicians agreed that it 
characterized the case. If all clinicians across the six sessions agreed 
on a diagnostic stat^mentj the proportional agreement would be 1,00. 

To examine agreement between two clinicians on diagnostic statements 
seen as 'characteriging a cases or for one clinician on a case and its 
replicatej correlation matrices ware constracted, A partial intra- 
clinician correlation matrix is presented in Table 1, A phi Qorrelation 
was calculated for each matrix (Appendix A) as well as a second statistic, 
the Porter correlation, developed by Andrew Portqr (Appendix A) , ' All 
inter- and intracorrelations are presented using the phi correlation and, 
next to it in parenthesis, when available, the Porter statistic. 



Table 1. 



Parcial Intracorrelatlon Matrix for Diagnosis 
Case 3, Clinician A, Time One, Clinician 
A, Time Two 



STATEMEmv4-+ 

Present: at Time Ong ^ and T ime 
Two - 

Motor coordiriatioh (w) 
Incallectual potential: 
General (S) 
^ Oral reading: Phrasing (W) 
Oral reading: Intonation (W) 



STATO^IENT 

Absent at Time One, Present 
at Time Two 

Motor coordination (Obs) ■ 
Hearing acuity (W) 
Speech articulatidm (W, Oba) 
Attitude toward reading: 

Independent (W)'. 
Attitude toward reading* 

Instructional (W) 
Relationship to peers (W) 
Ability to apply reading 

skillB (W) 
Oral reading: General (Obs) 
Oral reading:^ Rate.(w) 
Oral reading: 

Self --correct ion (W) 
Silqnt readingf General (W) 
Word analysis: General (S) 
Phonetic analysis: General (Obs) 
Use of initial consonant 

sounds (S) 
Use of syllables (S) 
Word recognition:. General (S) ^ 
Comprehensions Oral (S) 
Comprehensions Listening (S) 



STATEMENT +^ 

^regent at Time One, Abse nt at 
Wme^Two ~ . 

Attitude toward reading: 

Independent (Obs) 
Motivation for reading (Obs) 
Emotional adjustment (W) 
Substitutions contextually 

acceptable (W) 
Silent reading comprehension (W) 
Word analysis (W) 
Phonetic analysis (W) 
Comprehension vocabulary (Obs) 



STATEMENT \_ ^ ^ 

Absent .Both at Time One and Time 

Two " ^ ' ~~ ■ 

272 Domain. statements excluded 



Resulcs 



The results o( Cha 1977 Observational Scudy (VinHonhaler , Note 3) wich 
respect to group agrdement showad that moat diagnostic stateTnencs were - 
made only once for a given case. Across^ cases, only six diagnostic 
stataments .were mentioned in three or more sessions. T^^^^se statements 
were: =(1) at least average reading potential, (2) -r oral reading, 
(3) sight words low, (4) phonics weak, (5) poor word analysis skills, 
and (6) auditory discrimination probl.^m. Despite lengthy individual 
diagnostic write-ups, the clinicians could agree on only a few statemants 
characterizing any given case. 

When the diagnostic statements of any two, clinicians on a given case 
ware compared, the results showed that on the average, they agreed on 
virtually no diagnostic statements. When the diagnostic statemente 
across two cases (case/replicate) for a single clinician were compared, 
it was. found that, on the average, fewer than one quarter of the stttements 
mentioned by each clinician the first time s/he diagnosed a case were 
repaated when s/he diagnosed the replicate of the case. 

The diagnostic agreement for the specialists in the 1977 observational 
study is summarized in Table 2. 



Table 2. 



Mean Diagnostic Agreement of Reading Specialists 



Inter cor relations 



Intracorrelations 



Phi 



Porter 



Phi 



Porter 



Observacion Study, 



-0.07 



(p. 00) 



(0.23) 



1977 



The unexpactedly low diagnoscic agreement results in this study were 
scartlirig, parclculariy since the clinicians wht) participaced were 'highly 
trained (all but two had doctoral degrees) and had an average of 10 
years experience in their field. Clearly, before any conclusions ot 
generalizations could be drawn, a numbav o. possible explanations for the 
results had to be ruled out. The Clinical Studies group tested the validity 
C3f seven of these possible explanations ? or hypotheses. 

Hypothesis X , 
The Findings Were Low Due t o tihe Natur e o f the 
Training R eadl^ng Clinic ians Received 

• Perhaps the clinicians were not adequately prepared to be consistent 

diagnosticians. ^ Additionally, perhaps the sample was an unrepresentative 

one, and another group of similarly trained specialists would perform 

more reliably* 

In order to test these hypotheses, a second observational study was 
conducted (VanRoekel, Note 4). Twenty learning disabilities clinicians and 
20 reading clinicians diagnosed two simulated cases; One was a child 
with learning as well as reading disabilities; the other was a child with 
a reading disability only.' If differences in training were indeed a key 
factor in performance, Chen the learning disabilities specialists could 
be expected to show gr* r agreement on the learning disability caseg 
while the reading specialists should show greater agreement on the reading 
case* 

The group agreement results paralleled those of the 1977 study. 
Despite lengthy Individual diagnostic write-ups, a very small number of 
statements were agreed-upon as characterising the learning dlaabilities 
case (weakness in gross/fine coordination; problem with visual perception/ 
discrirainatidn/meraory/motor skills) and the reading case (average 
intellectual potentlall problem^ with attitude/ interests; weak phonic 



analysis skills; observations about contextual reading ability). 

The performance of the two groups, of clinicians reveale.1 no 
diffarential training effect whatever. Both groups performed at 
a naar zero level of, reliability, even within their o^^ area of 
specialization (Table 3). 



Table 3, 



Mean Diagnostic Agreement of Reading and 
Learning Disabilities Specialists 



inter cor relations 



Reading Case 



Learning 
D'lsabillcy Case 



Observational 
Study of 
Reading and 
Learning 
Disabilities 
Specialists 



Reading Learning ReadinR Learning 

Specialists Disability Specialists Disability ^ 

Specialists Specialists 



0.06 0.04 
• X = 0.05 



0.01 0:07 
X = 0.04 



In sum, an average of only 5% of the diagnos.tic statements made for a 
case could be agreed-upon by any. two clinicians examining that case. 

These figures very nearly duplicated the intercorralations reported 
for the 1977 study, and it was therefore felt that unique training effects 
and sampling error could be ruled out as explanations for the low diag- 
nostic agreement of the reading specialists. ■ ' 



- Hypotheals 2 . ^ 
The Findings Wer e Low Due to the^Nature of Clinical 
Tralnlrig In Reading and Learning Disabilities 

Perhape the training programs, Both in reading and learning disa- 

• bilities/ were deficient In that they did not provide sufficient opportunity 

fbr the clinicians to Imolenient .their diagnostic findings it. classroOTn 
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sdctinga where they couid gat feedback about the accuracy ot their diag- 

noscic Judgments. Classroom teachers, on the other hand, trained In using 

test- teaching in classroom settings might make m.ore reliablta diagnostic 
Judgments. 

This hypothesis was tested in a third observational study (Gil, 
Note 5). Ten classroom teachers participated in this study, five from the 
Lansing, Michigan area and five from the Chicago, Illinois area. Two 
classroom-oriented simulated cases were developed. The Chicago teachers 
had been trained to perform diagnoses using only materials normally 
available in a classroom; the Lansing teachers had not received such 
training. 

Once again, only a small portion (6%) of total diagnostic statements 
made by the group were agreed upon as being characteristic of the cases-. 
(1) poor comprehension, (2) knows toajor vocabulary concepts, (3) sight 
words weak, (4) ignores endings, (5) sight vocabulary good, (6) phonic 
skills weak, (7) problems with oral reaaing, and (8) word attack skills. 

The extent of agreement betwaen t^o teachers on diainostle statements 
was affectively zero. There was no difference between teachers who had 
been trained in techniques of clansroom diagnosis and those who had not 
(Table 4), 



Table 4. 



Mean Diagnostic Agreement of Classroom Teachers 



Intercorrelations 



Observational Study of 
Classroom Teachers 



Case 7 



- 0,04 



Case 8 



- 0,03 



Once again, this findings Teplicated thusa of aarlier BCudius: (1) a 
meager consensus on scatements characterizing a case was discerned only 
by aggregating diagnoses across clinicians; and (2) clinicians exhibiced 
extremely low levels agreement on the same case. Thus, ostensible 
differences in training programs did not result in differences in per-- 
formance for che reading and learning disabilities specialists and class-- 
room ceachers investigated thus tar. Perhaps these subjects did not 
raliably diagnose the cases because no program trained them to do so. 
However, other possible eKplanations for che low findings needed to be 
pursued. 

Hypothesis 3. ' 
The Findings Were Low Due to a Lack of Standardized 

Vocabul ary Among the Clin i cians ^ ' , 

Perhaps in categorizing the clinicians- natural language statements, 
the experimenters failed to see equivalences. In that , case, statements 
that were actually describing the same, thing would be coded as being 
dissimilar^ and , agreement would appear to be very low* 

A fourth observational study was undertaken (Hoffmeyer, 1980). This 
study was designed to replicate the Initial 1977 investigation^ adding 
Che use of a standardized diagnostic check! empirically derived from 
clinician-,s statements in the preceding observational studies* The 
reading clinicians transferred their o\m natural language statements to 
the standardized checklist. All analyses were based on the checklists, 
thereby eliminating all coder subjectivity with respect to equating 
natural language statements. 

Group agreement across cases focused on the same diagnostic categories 
as the original investigations^ (1) at least average Intellectual poten-.. 
tial, (2)' poor oral reading, (3) sight words low, (4)' phonics weak, 
(5) poor word analysis skills, and (6) auditory acuity.^ There was some 

1 - ' 



■ . - • '. . . ■ "10 

agreement on two additional categories^ problem with, comprehension and 
poor attitude toward reading, for any^ given case, then, oniy 3 ,few 
statements couid be gleaned which represented group agreement on case 
chara^cteristics* ' ^ ^ ^ " 

. The results of comparing the diagnostic statements of any two 
cllni'cians for m ease showed "a slight increase over the original study. 
The performance of one clinician over time remained essentially the same 
(Table 5), . ' . ^. 





Table 5. . \ ^ 


Mean Diagnostic 


Agreement of Reading Specialists 




Diagnostic Agreement ^ 




Intercorrelations Intracorrelatlons 




^ Phi Porter I ' Phi Porter 


1978 Replicate using. 




Standardised Diagnostic 




Checklist 


0,11 .(Oai) , .0,25 (0,17) 


ThuSj differences in 


vocabulary did not appear to change clinician 


reliability to any aignif leant degree. 




Hypothesis 4, 


The Findings 


Were Low Due to the Nature 


of the 


Experimental Setting ^ ' > • 



a Perhaps the requfrement that informatJ.on be requested item by item 

/was tqo dissimilar froi^ the conditions under which clinicians actually 

i = I- ' ' • 

w,ent dbout t'he task of diagnosing reading difficulties, 

A fifth observational study '"(Stratoudakls, Note 7) was carried 'out 

in which the simulate^case format was altered. The amount of case ^ i ' 

//ranterial was^reducedj and the information was presented in a three-ring 

notebook. ;Inltead^of requesting itfimaSf information from an experirainter 



the clinicians worked Independently with^^U ^ the cifes at hand in the note-'' 
book; (This .foraat had' the additibhal ^advantag'e of being more pdonomicali 
tawer items of information redaced mfiterlals .cogts^ a'nd an ex^Brimenter 
was no Ipnger required.) ^ " ^ ' ^ ' ' ' \ 

Diagnostic agreement both between and within trinicians' replicated 
that founa 4h earlier studies tTable 6) ^ Thus^ diagnostic 'a|reem^ * 
remain|ed virtually unchanged despite, the altered-^ forakt and procedures. 



T 



Tablb 6. ^ a 



Heart Diagnostic Agree^ieat Using No tebpok,'^ Format , ^ , . / 

. . " ^ ^ ^ DiagnSs'tic ^Xgreement 

* ^ - ■ ^ * . — " 

' tnte'^orr elaHons ' ^ ' . Intracorreiations 

I o ^ J * Ihi ^ PSEMr ' ' ^ ^ Phi ^ Port er 

■ ' ^ ^ ' t ^ — " * ' — 

Observational Sthdy Using^ ^ ^ / " " v 

Notebook Format ' . 0.^13 j^K^O. ip) *v: ^0^20 (0.12) 



A final. set of hypptheses was formuiated in an attemptr^to -account '¥or 
the low diagnostic findings that ha& continued to be replicated through . 
the preceding studies/ ^ Thefe hypo^he^es were^ tffeted^ J.n a final pbserva- ' 
tional study that emphasized the relationship bttw^en didgnofils and 
remediation* ' : - ' , - 

' * Hypothesis- 5 , ' < ' - 
The Findings Were Bow Due-^ to the* Inclusion ^ ^ ■ < ' ' 
' of Ptgcrlptive Diagriostdc Statements - ^ ^ 

! - = ^ ■ . ' ' . * ' ^ -» * 

^Perhaps ,we' in the Clinical Studies. group were overlooking a core" . - 

group of diagnostic^ statements to which reihedi^tlotis wete consletently 

attached. By uairig as the unit of , analysis, all dlagnbstUc statements 

made ligstead :'Bf ffSt those which ^were seen ^s needing remediation^ we 

might'^ave ir^dver tently "swamped" *aubstantial agreement on remediatld 

diagnostic statements* . ' • * v ' ' . . 



* Hypothesis 6 , 

* \ The Findings Were Low Becauae the Che'ckllst 
' . ' 4Jsed in the '^1978 RepllGatlon ^Stiidy " • ' ^ . '' 

Might Have Been Excessively CQmplex 

^ , A shorter more fiightly organiEed' checklist derived from that earlier 

' study might make the translation process more accurate. - ^ * 



• ■ * Hypothesis 7 . 

The Findings Were Low Begause ^Diagnostic Rellabirity ^ 
Is Not Important to Clinicians Bu t Remadlal 
Reliability Js and Will Be Reflected in Greater 
"Reliabliity with Respect to Actions Chosen - 

The-final observational study (Welnshaak, Note 8) in this three-year 
series of investigations addressed Hypoth^s. 5, 6, and 7. In this studyj 
the eight experienced reading clinicians, ^ fo'lir trained in Michigan and four 
in Illinois, transferred their diagnostic statements to a shortened diag-'; 
ndstt^ ehef:kllst, their remedial atateraents to an jsmpirically derived 
ranedlal checklist, and eKplicljtly associate^ remedial and dlagnostia 
statements. 

Yet again, a smali portion (101) of diagnostic categories menttioned 
aficounted for whatever group agreeraant existed across cases! (1) at ' 
least average intellectual potential, and problems with (2)' word recog- 
nition, (3) word 'analysis, (4) oral reading, (5) s.ilent reading, . (6) . 
comprehension, (7) auditory/visual acuity, (8) auditory discrimination, 
and (9) affect. The .results are .sunmiarized i^n Table '7, 



^ . Tabid 7. . 

Mean Agreement for DiagnoaiH T Remediation, nnd RemtHllntid DliignoHUH 

Agrooment ' ' 



Int ercorre 1^1 1 Ions 



I ntracpr CO la t iiuiH 







Phi 


Porter 


Pill , 


PortiT 


Observational 


Diagnosis " 


0. L6 


(oai) 


0. 2i ' 


•• to. i4), 


Study Emphasizing 










Relationship Be- 


Remediaclon 


n. U 


(oao) 




(0.:>0) 


tween Diagnosis and ^ 












Remediation^ 


Remediated 


0. 13 


(O.OH) , 


0.22 


(0. 14) 




Diagnoses 
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The results showed^that (1) globnl diaKnoHtit^ rtOlainiLLy rtMiiaJnt^d 
unacQeptfably Ibw; (2) agreement on remedial nctidnH* Ln l:u uhckI fcirml 
^i^ually poOT'ly; and (3) agreement on prociHuly which dla^nos^'H warranLcHl 
treatment fell lower still, . /■ 

ThuSj none of the final three hypotheHua was supportLul by tlu* datn 
from thlB 3tud)^* . 1. 7. . . ^ 

Cpnclusi ons ^ 
Low dlagnos^tic (and^ remedial) re 11 ah 1 11 ty for renci I rig 'wnHl lt*arn I nj* 
disabilities specialists and claHsroom tbacherH appt^arH to be* a rcihnni ' 
phenomenon (Table 8), Across HtudleSj tbu' mean agrt'umt-nL bOlwovn nny L'wu 
tlinlcians on, a given case was Q, 08 (the range was 0.00 to 0.16)* Tht'y 
agreed, on the average^ on only 8% of thulr combined HLaLumfintH, whf<ili Im 
no better than the agreement expected due. to ch/|nce, Thn me/in agrcumcriL" 
across studies for a single clj nlc Ian . on a case and Itn repll<'/iLu wan 
0,20 (the range was 0,13 to 0,25), .Given the Identlcn! nnHe on twcj 
separate occasions, -a given clinician^ on the hveragei agrt*ed wl bh hlfn/hi'r-- 
' self on only 20%]of the combined sCaLamanta for the* cnHe and i lh rcnllfinic 
Why are these traiTied .and exper I cinced profoHHlnna 1 h .por rorm J riH 
unreliably? Can their dlagnoHLic and remed lal ^ rel lab! ] i Ly hi* jmprc»vi'dV 



Further analyses' of individual performance in the studies reported here 
seemed to cdnflrm chat the chief cause ^f or low diagnostic and remedial 
reliability in reading ^as inadequate or inappropf ia^te Craining 
tyinsonhaler, Note 3), „ 

/Short-term (30 instructional hours) training studies conducted by ' 
the Clinical Studies group ' (Sherman, Welnshank, h Brown, Note 9; Gil, ' ' 
Polins Vinsonhalers & VanRqekel, Notei 10) have pilot^tested procedures 
which doubled the average entering ^reliability of those who w^ere trained. 



" Table' L 

^ Diagnostlc,^Relia.bility of Reading Specialists , ^ 
Leaining Disabilities Specialists and Classroom Teachers 

Jntercorrelations Intracorrelatlons 



.Phi Porter Phi Porter 

Observational Study, 1977 " -o!q7 (0.00) oil3 (0,23) 

Observational Study of Readrng COS — ' 

and Learning Disability (reading case) 
^ Specialists , ' 

r - ■ ^ 0-04 ' ' . 

^ ' (^earning' Disa- 
> . . . bility Case) 

Obs ervational Study of Classroom '^0.04 — ^ 

Teachers/ " • ^ ^(Case 7> ' 

^"0.03 . . ~ ^ . 
Cfiase 8) . , 

Observatibnal Study 1977 0,11^ (Cll) 0.25 C0.J7) 

Replication' / ' ' y. / * . 

Observational Study Using 0,13 (O.'IO) 0.rr20 - (0,12) 

Notebook Format ' ^ . - ' 

Observational Study Emphasizing 0,16 (0*11) 0.23 ^ C0.14) 

Relationship Between .Diagnosis ^CDlagnoa'ls). > 
and Remediation Plan ' ' ^ ' m , ' 

. 0.14 (OaO) \ 0.29 (0.20X 

, / . ; (Remediation) 

_ ■ ^ . ' , 0.13 • (O.O80 0.22- ' (0.14) ' 
' , (Remediated diagnoses) 



15 

, The- training included : (1) the use 6"f a modal-based procedure to guide 
the tasks of dlagnosds and prescription, '(2)t direct draining In the use 
of *declsioa aids arid standard ^vocabulary, and (3) extended practice with' 
feedback on diagnostic performance* 

These studies have not, however , ruled out other possible sources of 
Improvement, nor have they determined how,^ or whether, mora substaotial 
gains might be achieved. Research now in progress in the Clinical Studie^^^ 
Project is addressing these questions. Additionally, the validity of 
model-directed diagnosis and remediation is being studied in the context 
of , systematic classroom-based follow-up of children with reading diffi- 
culties. Taken together, these lines of inquiry are likely to have sig- 
nificant' implications for preservice teacher training, advanced clinical ^ ' 
trainlngj and inservice professional deveropment programs. » 
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APPENDIX A 

Calcdlation of Phi. Correlation^ 
and Porter Statistic 



M 



c 

a 
c 



D 

S 
O 



Qlinician 1 SIMCASE Q, Fom One' 
PRESEOT (+) ABSENT (^). 



Frequency count of 
statements present in 
the domain in both ^ 
Sessions for form one 
and form twb of SIM- 
CASE 



Frequency cbunt of. 
statements in the 
rdomaln present in 
the session for SIM-^ 
CASE form one but not 
SIMCASE form two 

\ ^ C 



Frequency count of 
statements, present in 
the (Jomain pre^«:*nt in 
SIMCASE form two but 
not^ £n SIMCASE form 
one 

B 



Absent in both 
sessions for form 
onej and form two of 
SI>iCASE 









a {++) . 


b (+-) 


a'+ b 


; c (-+) 




c + d 






i + c 


1 -- -- — - 

b + d 


N 



Phi; = 



(a X d - ^b X c) 



• (a + C) X (b + d) X (c +^d) X (a + b) ' 

The presence of a large percentage ol statements (more than' 85%) 
in the VD" pell (the statement is absent in bdth sessions) artif iciariy . 
inflated tha intercorrelations sinfce it represented, in effect, agreeing 
to disagree, A statistic developed by Professor A. Porter (Institute 
for Research on Teaching, Michigaii State University) was designed- to 
correct for this occurrence, by including^in the computation only the 



Values,.in the A, B, and C cells ^ 

, • , ■ ■ t - I A + B ^• c 



